Head mounted display apparatus

ABSTRACT

When a hand of the user is recognized in an image pickup region of a camera, a head mounted display monitors behavior of the hand in the image pickup region. When the hand of the user in the image pickup region reaches an outer peripheral region forming an outer periphery of the image pickup region, a notification is give to the user.

BACKGROUND

1. Technical Field

The present invention relates to a head mounted display apparatus.

2. Related Art

A technique called Augmented Reality (AR) for adding information to areal environment and presenting the information using a computer isknown. A technique for realizing such augmented reality is applied tonot only head mounted display (HMD) apparatuses (for example, refer toJP-A-2010-146481) but image processing devices (for example, refer to“Handy AR: Markerless Inspection of Augmented Reality Objects UsingFingertip Tracking”) which display an image on the existing displaysother than HMDs. The HMD disclosed in JP-A-2010-146481 allows the userto visually recognize a virtual panel formed by augmented reality as avirtual image, picks up images of a position and a movement of a hand ofthe user with a camera, and associates image pickup positions thereofwith the virtual panel. Therefore, a technique for recognizing the handof the user is necessary, and such hand recognition techniques aredisclosed in “Handy AR: Markerless Inspection of Augmented RealityObjects Using Fingertip Tracking” and “The international Journal ofVirtual Reality, 2009, 8(2): 7-12 Robust Hand Tracking Using a SimpleColor Classification Technique”.

For example, when a hand of the user is detected by a camera picking upan image thereof as disclosed in JP-A-2010-146481, currently, the usermoves the hand based on past experience and a projected virtual image.Therefore, there is a problem in that an image of the hand which hasbeen detected cannot be picked up by the user moving the hand.Therefore, in HMDs to which augmented reality (AR) is applied, it hasbeen required that a hand of the user be continuously detected. Inaddition, improvement in the versatility of a technique for detecting ahand of the user, reduction in cost thereof, and the like have been alsodesired.

SUMMARY

An advantage of some aspects of the invention is to solve at least apartof the problems described above, and the invention can be implemented asthe following forms.

(1) An aspect of the invention provides a head mounted displayapparatus. In this head mounted display apparatus, a virtual image andan outside scene overlap each other and can be visually recognized by auser, the head mounted display apparatus including: an augmented-realityprocessing unit that generates image data for displaying the virtualimage and allows the user to visually recognize the virtual image suchthat the virtual image is displayed in a visual field of the user basedon the generated image data; a detecting unit that detects a hand of theuser in a predetermined detection region for detecting the hand of theuser and detects behavior of the hand of the user in the detectionregion; and a notifying unit that gives a notification to the user basedon the behavior of the hand of the user detected by the detecting unitwhen the hand of the user reaches an outer peripheral region forming anouter periphery of the detection region.

In the head mounted display apparatus of this aspect of the invention,when the user moves and puts the hand into the detection region, andwhen the hand of the user is about to be moved from the detection regionto the outside thereof, the notification is given to the user who movesthe hand. The user who receives this notification can recognize that, ifthe hand is moved any further, the hand would deviate from the detectionregion. Therefore, the user can stop the movement of the hand or canreturn the hand to the detection region such that the hand is held inthe detection region. As a result, according to the head mounted displayapparatus with this configuration, once the hand of the user isdetected, this detected state can be continued, which contributes toimprovement of the detection accuracy of the hand.

(2) In the head mounted display apparatus according to the aspect of theinvention described above, the augmented-reality processing unit mayallow the user to visually recognize the virtual image in a displayregion corresponding to the detection region, and when the detectingunit detects the hand of the user in the detection region, theaugmented-reality processing unit may generate the image data fordisplaying the virtual image, which is embedded with a partial handimage corresponding to at least a part of the detected hand of the user,and may update a position of the partial hand image in the displayregion of the virtual image based on a position of the part of the handin the detection region. As a result, since the user can visuallyrecognize the virtual image including the partial hand image, the usercan recognize a correspondence between the hand of the user and thepartial hand image of the virtual image.

(3) In the head mounted display apparatus according to any of theaspects of the invention described above, the augmented-realityprocessing unit may generate the image data for displaying the virtualimage, which is embedded with a frame shape corresponding to the outerperipheral region forming the outer periphery of the detection region,to allow the user to visually recognize the virtual image. As a result,the position of the hand of the user in the detection region can bereliably recognized by the user. Therefore, the detected state of thehand of the user can be continued and the detection accuracy of the handcan be improved with high effectiveness.

(4) In the head mounted display apparatus according to any of theaspects of the invention described above, when the detecting unitdetects the hand of the user in the detection region, theaugmented-reality processing unit may generate the image data fordisplaying a detection-completed image, which indicates the completionof the detection of the hand of the user, as the virtual image or maygenerate the image data for displaying the virtual image, which isembedded with the detection-completed image, to allow the user tovisually recognize the virtual image. As a result, when the hand ismoved around its current position, the user can recognize that the handis held in the detection region. Therefore, once the hand of the user isdetected, this detected state can be continued, which contributes toimprovement of the detection accuracy of the hand.

(5) In the head mounted display apparatus according to any of theaspects of the invention described above, the notifying unit may beconfigured to change a form of a notification to the user and, as thehand of the user in the detection region approaches the outer peripheryof the outer periphery region, may give the notification to the userwhile changing the form of the notification. As a result, when the handof the user in the detection region is moved outside the region, achange in the form of the notification can reliably allow the user torecognize this state and can urge the user to hold the hand in thedetection region. Therefore, the detected state of the hand of the usercan be continued and the detection accuracy of the hand can be improvedwith higher effectiveness.

(6) In the head mounted display apparatus according to any of theaspects of the invention described above, as the hand of the user in thedetection region approaches the outer periphery of the outer peripheryregion, the notifying unit may emit a sound with a higher frequency oremits a sound with a higher volume to give the notification to the user.As a result, when the hand of the user in the detection region is movedoutside the region, the sound can reliably allow the user to recognizethis state and can urge the user to hold the hand in the detectionregion. Therefore, the detected state of the hand of the user can becontinued and the detection accuracy of the hand can be improved withhigher effectiveness.

All the above-described plural components included in each aspect of theinvention are not essential. For a part of the plural components,modifications, deletions, substitutions with new components, and partialdeletions of the limitations can be appropriately made to solve a partor all the above-described problems or to achieve a part or all theeffects described in this specification. In addition, in order to solvea part or all the above-described problems or to achieve a part or allthe effects described in this specification, an independent aspect ofthe invention can be made by combining a part or all of the technicalfeatures included in one aspect of the invention with a part or all ofthe technical features included in another aspect of the invention.

For example, one aspect of the invention can be implemented as a headmounted display apparatus and can also be implemented as anotherapparatus other than the head mounted display apparatus. A part or allthe technical features of each configuration of the above-described headmounted display apparatus can be applied to this apparatus.

The invention can be implemented in various forms. For example, theinvention can also be implemented in forms of a control method for thehead mounted display apparatus, a head mounted display system, acomputer program for implementing functions of the method, theapparatus, or the system, a recording medium on which the computerprogram is recorded, and the like.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described with reference to the accompanyingdrawings, wherein like numbers reference like elements.

FIG. 1 is a diagram illustrating a schematic configuration of a headmounted display apparatus according to an embodiment of the invention.

FIG. 2 is a block diagram functionally illustrating a configuration of ahead mounted display.

FIG. 3 is a diagram illustrating a state where image light is emitted byan image-light generating unit of a right display driving unit.

FIG. 4 is a diagram illustrating an example of a virtual image which isrecognized by the user through augmented reality processing executed byan AR processing unit.

FIG. 5 is a diagram illustrating the summary of hand recognitionprocessing which is executed by a CPU including an image processing unitand the AR processing unit.

FIG. 6 is a flowchart illustrating the order of the hand recognitionprocessing.

FIG. 7 is a diagram illustrating an input status of image pickup data.

FIG. 8 is a diagram schematically illustrating a state where adifference between adjacent pixels of a part of a hand in an imagepickup region is calculated.

FIG. 9 is a diagram schematically illustrating the results of thedifference calculation and grouping in the entire region of the imagepickup region.

FIG. 10 is a diagram illustrating the summary of hand contours which arestored in a hand contour shape storing unit of a storing unit.

FIG. 11 is a diagram schematically illustrating a state where a virtualimage is embedded with a pointer corresponding a fingertip of arecognized hand of the user.

FIG. 12 is a diagram illustrating the summary of notification processingwhich is executed by a control unit including the image processing unitand the AR processing unit.

FIG. 13 is a flowchart illustrating the order of the notificationprocessing.

FIG. 14 is a diagram illustrating a relationship between behavior of thehand held in the image pickup region and an outer peripheral region anda relationship between the behavior of the hand held in the image pickupregion and an emission status of an alarm sound.

FIG. 15 is a diagram schematically illustrating hand recognitionprocessing which is executed by a head mounted display according toanother embodiment of the invention.

FIG. 16 is a diagram illustrating a schematic configuration of a headmounted display according to another embodiment of the invention.

FIG. 17 is a diagram schematically illustrating a relationship betweenan image pickup region of a camera, which is positioned at an end of animage display unit, and a visual field of the user.

FIG. 18 is a diagram illustrating the summary of a method of detecting ahand when the image pickup region of the camera deviates from the visualfield of the user.

FIGS. 19A and 19B are diagrams illustrating a configuration of anexternal appearance of a head mounted display according to amodification example of the invention.

DESCRIPTION OF EXEMPLARY EMBODIMENTS A. Embodiments A-1. Configurationof Head Mounted Display Apparatus

FIG. 1 is a diagram illustrating a schematic configuration of a headmounted display apparatus according to an embodiment of the invention. Ahead mounted display apparatus 100 is worn on the head and hereinafterwill also be referred to as “head mounted display 100”. The head mounteddisplay 100 according to the embodiment is an optically transmissivehead mounted display apparatus with which the user can visuallyrecognize a virtual image and can visually recognize an outside scenedirectly at the same time.

The head mounted display 100 includes: an image display unit 20 thatallows the user to visually recognize a virtual image in a state ofbeing worn on the head of the user; and a control unit (controller) 10that controls the image display unit 20.

The image display unit 20 is a wearing body that is worn on the head ofthe user and has an eyeglass shape in this embodiment. The image displayunit 20 includes a right holding unit 21, a right display driving unit22, a left holding unit 23, a left display driving unit 24, a rightoptical-image display unit 26, a left optical-image display unit 28, anda camera 61. The right optical-image display unit 26 and the leftoptical-image display unit 28 are arranged to be positioned before theright and left eyes of the user when the user wears the image displayunit 20. An end of the right optical-image display unit 26 and an end ofleft optical-image display unit 28 are connected to each other in aposition corresponding to the middle of the forehead of the user whenthe user wears the image display unit 20.

The right holding unit 21 extends from an end ER, which is the other endof the right optical-image display unit 26, to a position correspondingto the temple of the user when the user wears the image display unit 20.Likewise, the left holding unit 23 extends from an end EL, which is theother end of the left optical-image display unit 28, to a positioncorresponding to the temple of the user when the user wears the imagedisplay unit 20. The right holding unit 21 and the left holding unit 23hold the image display unit 20 on the head of the user as in the templesof glasses.

The right display driving unit 22 is arranged inside the right holdingunit 21, that is, on a side opposite the head of the user when the userwears the image display unit 20. In addition, the left display drivingunit 24 is arranged inside the left holding unit 23. Hereinafter, theright holding unit 21 and the left holding unit 23 will be collectivelyreferred to simply as “holding unit”, the right display driving unit 22and the left display driving unit 24 will be collectively referred tosimply as “display driving unit”, and the right optical-image displayunit 26 and the left optical-image display unit 28 will be collectivelyreferred to as “optical-image display unit”.

The display driving unit includes liquid crystal displays 241 and 242(hereinafter, referred to as “LCD”) and projection optical systems 251and 252 (refer to FIG. 2). The details of a configuration of the displaydriving unit will be described below. The optical-image display unit asan optical member includes light guide plates 261 and 262 (refer to FIG.2) and a dimming plate. The light guide plates 261 and 262 are formed ofan optically transparent resin material or the like and guide imagelight, which is output from the display driving unit, to the eyes of theuser. The dimming plate is an optical member having a thin plate shapeand is arranged to cover the front side (side opposite the eyes of theuser) of the image display unit 20. The dimming plate protects the lightguide plates 261 and 262 and suppresses damages to the light guideplates 261 and 262, adhesion of stains, and the like. In addition, byadjusting light transmittance of the dimming plate, the intensity ofexternal light entering the eyes of the user can be adjusted and theeasiness of the visual recognition of a virtual image can be adjusted.The dimming plate is not necessarily provided.

The camera 61 is arranged in a position corresponding to the middle ofthe forehead of the user when the user wears the image display unit 20.The camera 61 picks up an image of an outside scene (scene on theoutside) in a front side direction of the image display unit 20, thatis, in a visual field direction of the user wearing the head mounteddisplay 100. The camera 61 is a so-called visible light camera andincludes an imaging device such as a charge coupled device (CCD) or acomplementary metal-oxide semiconductor (CMOS). An outside scene imageacquired by the camera 61 is an image indicating the shape of an objectfrom visible light radiated from the object. In the embodiment, thecamera 61 may be a monocular camera or a stereo camera. In addition, aninstallation position of the camera 61 is not limited to the middle ofthe forehead of the user and may be the end EL or the end ER of theimage display unit 20.

The image display unit 20 includes a connecting unit 40 for connectingthe image display unit 20 to the control unit 10. The connecting unit 40includes a main body cord 48 that is connected to the control unit 10, aright cord 42 and left cord 44 that are branched from the main body cord48, and a coupling member 46 that is provided in a branching point. Theright cord 42 is inserted from a distal end AP of the right holding unit21 in the extending direction thereof to the inside of a housing of theright holding unit 21 and is connected to the right display driving unit22. Likewise, the left cord 44 is inserted from a distal end AP of theleft holding unit 23 in the extending direction thereof to the inside ofa housing of the left holding unit 23 and is connected to the leftdisplay driving unit 24. The coupling member 46 is provided with a jackfor connecting a earphone plug 30 thereto. A right earphone 32 and aleft earphone 34 extend from the earphone plug 30.

The image display unit 20 and the control unit 10 transmit varioussignals through the connecting unit 40. An end of the main body cord 48opposite the coupling member 46 and the control unit 10 are providedwith connectors (not illustrated) which are fitted to each other. Thecontrol unit 10 and the image display unit 20 are connected anddisconnected by establishing and releasing the fitting between theconnector of the main body cord 48 and the connector of the control unit10. For the right cord 42, the left cord 44, and the main body cord 48,for example, a metal cable or an optical fiber can be adopted.

The control unit 10 controls the head mounted display 100. The controlunit 10 includes a lighting unit 12, a touch pad 14, a cross key 16, anda power switch 18. The lighting unit 12 notifies the user of theoperation state (for example, ON and OFF of a power supply) of the headmounted display 100 through a lighting state thereof. As the lightingunit 12, for example, a light emitting diode (LED) can be used. Thetouch pad 14 detects a contact manipulation on a manipulation surface ofthe touch pad 14 and outputs a signal according to detected content. Forthe touch pad 14, various types of touch pads such as capacitive,pressure-sensitive, optical touch pads can be adopted. The cross key 16detects a manipulation of pressing keys corresponding to up, down, left,and right directions and outputs a signal according to detected content.The power switch 18 detects a manipulation of sliding the switch toswitch a power status of the head mounted display 100.

FIG. 2 is a block diagram functionally illustrating a configuration ofthe head mounted display 100. The control unit 10 includes aninput-information acquiring unit 110, a storing unit 120, a power supply130, a radio communication unit 132, a GPS module 134, a CPU 140, aninterface 180, and transmitting units (Tx) 51 and 52. The units areconnected to one another through a bus (not illustrated).

The input-information acquiring unit 110 acquires a signal correspondingto an manipulation input to, for example, the touch pad 14, the crosskey 16, and the power switch 18. The storing unit 120 is configured of aROM, a RAM, a DRAM, a hard disk, or the like. The storing unit 120includes a hand contour shape storing unit 122 and a frequency map 124.Stored content of the hand contour shape storing unit 122 and storedcontent of the frequency map 124 will be described below. The powersupply 130 supplies power to the respective units of the head mounteddisplay 100. As the power supply 130, for example, a secondary batterycan be used.

The CPU 140 reads a computer program stored in the storing unit 120 andexecutes the computer program to function as an operating system (OS)150, an image processing unit 160, a sound processing unit 170, adisplay control unit 190, and an AR processing unit 142. The ARprocessing unit 142 is triggered by an processing start request from theOS 150 or an specific application to execute processing for implementingaugmented reality (hereinafter, also referred to as “augmented realityprocessing”). The details will be described below. The AR processingunit 142 corresponds to “the augmented-reality processing unit” in theappended claims.

The image processing unit 160 generates a signal based on content(image) input through the interface 180. The image processing unit 160supplies the generated signal to the image display unit 20 through theconnecting unit 40. The signal which is supplied to the image displayunit 20 varies depending on whether the content is analog content ordigital content. In the case of the analog content, the image processingunit 160 generates and transmits a clock signal PCLK, a verticalsynchronization signal VSync, a horizontal synchronization signal HSync,and image data Data. Specifically, the image processing unit 160acquires an image signal contained in the content. For example in thecase of a moving image, the acquired image signal is generally an analogsignal formed of 30 frame images per second. The image processing unit160 separates synchronization signals such as the verticalsynchronization signal VSync and the horizontal synchronization signalHSync from the acquired image signal to generate the clock signal PCLKusing a PLL circuit or the like according to a period of thesynchronization signals. The image processing unit 160 converts theanalog signal, from which the synchronization signals are separated,into a digital image signal using an A/D conversion circuit or the like.The image processing unit 160 stores the converted digital image signalin the DRAM of the storing unit 120 frame by frame as image data Data of(RGB data). On the other hand, in the case of the digital content, theimage processing unit 160 generates and transmits the clock signal PCLKand image data Data. Specifically, in the case of the digital content,since the clock signal PCLK is output in synchronization with an imagesignal, the generation of the vertical synchronization signal VSync andthe horizontal synchronization signal HSync and the A/D conversion ofthe analog image signal are unnecessary. The image processing unit 160may execute image processing including resolution conversion processing,various color tone corrections such as adjustment of brightness andsaturation, and keystone correction on the image data Data stored in thestoring unit 120.

The image processing unit 160 transmits the clock signal PCLK, thevertical synchronization signal VSync, the horizontal synchronizationsignal HSync, and the image data Data stored in the DRAM of the storingunit 120 through the transmitting units 51 and 52. The image data Datatransmitted through the transmitting unit 51 will also be referred to as“image data for right eye Data1”, and the image data Data transmittedthrough the transmitting unit 52 will also be referred to as “image datafor left eye Data2”. The transmitting units 51 and 52 function as atransceiver for serial transmission between the control unit 10 and theimage display unit 20.

The image processing unit 160 includes a hand detecting unit 162 toparticipate in recognizing a hand of the user in addition to theabove-described processing. Specifically, the hand detecting unit 162 ofthe image processing unit 160 receives an input of image pickup dataobtained from each pixel of the camera 61 and executes various processesin the order of the hand recognition processing of FIG. 6 describedbelow, the processes including: calculation of a difference betweencolors of adjacent pixels represented by the image pickup data; thecapturing of a shape formed by sets of image pickup data in which thedifference between colors of adjacent pixels is within a predeterminedthreshold; and determination on whether or not the captured shape is theshape of the hand of the user. Accordingly, the hand detecting unit 162constitutes “the detecting unit” in the appended claims during the handrecognition processing of FIG. 6.

The display control unit 190 generates control signals for controllingthe right display driving unit 22 and the left display driving unit 24.Specifically, the display control unit 190 individually controls,according to the control signals, ON/OFF of driving of the right LCD 241by a right LCD control unit 211, ON/OFF of driving of a right backlight221 by a right backlight control unit 201, ON/OFF of driving of a leftLCD 242 by a left LCD control unit 212, ON/OFF of driving of a leftbacklight 222 by a left backlight control unit 202, and the like tothereby control generation and emission of image light by each of theright display driving unit 22 and the left display driving unit 24. Forexample, the display control unit 190 causes both of the right displaydriving unit 22 and the left display driving unit 24 to generate imagelight, causes only one of the display driving units to generate imagelight, or causes neither of the display driving units to generate imagelight. In addition, the display control unit 190 transmits the controlsignals for the right LCD control unit 211 and the left LCD control unit212 respectively through the transmitting units 51 and 52. The displaycontrol unit 190 transmits the control signals for the right backlightcontrol unit 201 and the left backlight control unit 202 respectivelythrough the transmitting units 51 and 52.

The sound processing unit 170 acquires a sound signal included in thecontent, amplifies the acquired sound signal, and supplies the soundsignal to a speaker (not illustrated) in the right earphone 32 connectedto the coupling member 46 and a speaker (not illustrated) in the leftearphone 34 connected to the coupling member 46. For example, when aDolby (registered trademark) system is adopted, sound signal processingis performed and different sounds having, for example, variedfrequencies or the like are output from the right earphone 32 and theleft earphone 34, respectively.

The interface 180 connects various external apparatuses OA, which aresupply sources of the content, to the control unit 10. Examples of theexternal apparatuses OA include a personal computer, a cell phone, and agame terminal. As the interface 180, for example, a USE interface, amicro USB interface, or a memory card interface can be used.

The image display unit 20 includes the right display driving unit 22,the left display driving unit 24, the right light guide plate 261functioning as the right optical-image display unit 26, the left lightguide plate 262 functioning as the left optical-image display unit 28,the camera 61, and a 9-axis sensor 66.

The 9-axis sensor 66 is a motion sensor for detecting acceleration(three axes), angular rate (three axes), and geomagnetic fields (threeaxes). The 9-axis sensor 66 is provided in the image display unit 20 andthus functions as a movement detecting unit for detecting movement ofthe head of the user when the image display unit 20 is worn on the headof the user. The movement of the head includes changes in the speed, theacceleration, the angular rate, and the direction of the head.

The right display driving unit 22 includes a receiving unit (Rx) 53, theright backlight (BL) control unit 201 and the right backlight (BL) 221that function as a light source, the right LCD control unit 211 and theright LCD 241 that function as a display element, and a right projectionoptical system 251. The right backlight control unit 201, the right LCDcontrol unit 211, the right backlight 221, and the right LCD 241 willalso be collectively referred to as “image-light generating unit”.

The receiving unit 53 functions as a receiver for serial transmissionbetween the control unit 10 and the image display unit 20. The rightbacklight control unit 201 drives the right backlight 221 based on aninput control signal. The right backlight 221 is a light-emitting bodysuch as an LED or an electroluminescence (EL). The right LCD controlunit 211 drives the right LCD 241 based on the clock signal PCLK, thevertical synchronization signal VSync, the horizontal synchronizationsignal HSync, and the image data for right eye Data1 which are inputthrough the receiving unit 53. The right LCD 241 is a transmissiveliquid crystal panel in which multiple pixels are arranged in a matrixshape.

FIG. 3 is a diagram illustrating a state where image light is emitted bythe image-light generating unit of the right display driving unit 22.The right LCD 241 drives liquid crystal corresponding to positions ofthe pixels arranged in a matrix shape to change the transmittance oflight transmitting through the right LCD 241, thereby modulatingillumination light irradiated from the right backlight 221 intoeffective image light representing an image. In the embodiment, abacklight system is adopted, but image light may be emitted using afrontlight system or a reflection system.

The right projection optical system 251 is configured of a collimatelens that changes image light emitted from the right LCD 241 into lightbeams in a parallel state. The right light guide plate 261 functioningas the right optical-image display unit 26 guides the image light outputfrom the right projection optical system 251 to the right eye RE of theuser while reflecting the image light along a predetermined opticalpath. As the optical-image display unit, an arbitrary system can be usedas long as a virtual image is formed before the eyes of the user usingthe image light. For example, a diffraction grating may be used or asemitransparent reflection film may be used.

The left display driving unit 24 has the same configuration as the rightdisplay driving unit 22. That is, the left display driving unit 24includes a receiving unit (Rx) 54, the left backlight (BL) control unit202 and the left backlight (BL) 222 that function as a light source, theleft LCD control unit 212 and the left LCD 242 that function as adisplay element, and a left projection optical system 252. The rightdisplay driving unit 22 and the left display driving unit 24 areprovided as a pair. Since the units of the left display driving unit 24have the same configurations and functions as the above-described unitsof the right display driving unit 22, the description thereof will notbe made.

FIG. 4 is a diagram illustrating an example of a virtual image which isrecognized by the user through the augmented reality processing executedby the AR processing unit 142. The image light guided to both the eyesof the user wearing the head mounted display 100 as described aboveforms an image on the retinas of the user. As a result, the user canvisually recognize a virtual image VI. As illustrated in FIG. 4, thevirtual image VI is displayed in a visual field VR of the user wearingthe head mounted display 100. In a portion of the visual field VR of theuser where the virtual image VI is displayed, the user can visuallyrecognize the virtual image VI of the optical-image display unit and canvisually recognize an outside scene SC, which is positioned behind thevirtual image VI, through the virtual image VI. In a portion of thevisual field VR of the user other than the portion where the virtualimage VI is displayed, the user can directly see the outside scene SCthrough the optical-image display unit. Image data for displaying thevirtual image VI and the outside scene SC which overlap each other asdescribed above is generated as image data representing additionalpresentation information for augmenting the outside scene SC recognizedby the user, through the augmented reality processing which is executedby the AR processing unit 142 of the head mounted display 100. The imagedata generated by the AR processing unit 142 is transmitted to the rightLCD control unit 211 and the like, and the virtual image VI is displayedon a front region of the user. “Augmenting the outside scene SC” impliesadding, deleting, emphasizing, and attenuating information regarding areal-world, that is, regarding the outside scene SC to augment theoutside scene SC which is the real world recognized by the user. Duringthe augmented reality processing for generating the image data, the ARprocessing unit 142 generates the image data for right eye Data1 and theimage data for left eye Data2, which are different, to embed the outsidescene SC with the additional presentation information. “Embedding theoutside scene with the additional presentation information” impliesdisplaying the virtual image VI on a position, which is distant from theuser by a predetermined distance, of the outside scene SC recognized bythe user, the virtual image VI making the user feel as if the additionalpresentation information is present. For example, if the virtual imageVI visually recognized by the user in FIG. 4 is an apple, image datarepresenting this apple image data which overlaps a real road includedin the outside scene SC is generated through the augmented realityprocessing, and an image based on this generated image data is displayedas the virtual image VI. As a result, the user can feel as if the applefalls on the road where there is nothing, and the AR processing unit 142generates, through the augmented-reality processing, the above-describeddata for right eye and data for left eye for displaying the virtualimage VI illustrated in FIG. 4 or the virtual image VI representing theapple on the real outside scene SC to be distant from the user by apredetermined distance and outputs the data.

A-2. Hand Recognition Processing

FIG. 5 is a diagram illustrating the summary of hand recognitionprocessing which is executed by the CPU 140 including the imageprocessing unit 160 and the AR processing unit 142. As illustrated inFIG. 1, the head mounted display 100 includes the camera 61 in the imagedisplay unit 20. Since the camera is positioned in the middle of theforehead, an image pickup region CR of the camera 61 occupiessubstantially the center of an information region of the visual field VRas illustrated in FIG. 5. The user recognizes an approximate positionoccupied by this image pickup region CR based on the camera position andthus can put a hand Yh thereof into the image pickup region CR by itsown will. When the user puts the hand Yh into the image pickup regionCR, the camera 61 picks up images of the outside scene SC including thehand Yh and outputs image pickup data corresponding to the individualpixels to the CPU 140. In the case of the outside scene SC of FIG. 5,when the user changes a point of view, a flying bird or a vehicletravelling the road may enter the image pickup region CR. In addition,for example, when the user wears the head mounted display 100 in a room,a table, a chair, a pet dog or cat, the face of a friend, and the likemay enter the image pickup region CR. The camera 61 picks up images ofthe above-described objects and outputs image pickup data correspondingto the individual pixels to the CPU 140. The head mounted display 100according to the embodiment determines whether or not an object enteringthe image pickup region CR is the hand Yh of the user as follows. FIG. 6is a flowchart illustrating the order of the hand recognitionprocessing.

This hand recognition processing is repeatedly executed, and the imageprocessing unit 160 receives an input of image pickup data per pixelincluded in the camera 61 (Step S100). FIG. 7 is a diagram illustratingan input status of the image pickup data. As illustrated in FIG. 7, inthe embodiment, the image pickup region CR is vertically andhorizontally divided into four divided regions CR11 to CR22, and thesedivided regions CR11 to CR22 are set as the unit of data input.Moreover, data is input for each divided region while being scanned foreach divided region in the horizontal direction by setting an upper leftpixel of each divided region as a data input start point and setting alower right pixel of each divided region as a data input end point. Suchscanning input of data is executed in the following order: dividedregion CR11→CR12→CR21→CR22. When receiving a scanning input of the imagepickup data, the image processing unit 160 cancels movement of the headof the user such as tilting or twisting based on sensor outputs from the9-axis sensor 66. While receiving the scanning input of the image pickupdata for each divided region along rows of pixels, the image processingunit 160 calculates a difference in pixel value between colors ofadjacent pixels represented by the input image pickup data (Step S102).FIG. 8 is a diagram schematically illustrating a state where adifference in pixel value between adjacent pixels of a part of the handin the image pickup region CR is calculated.

After the calculation of the difference in pixel value between adjacentpixels, the image processing unit 160 sets a set of image pickup data inwhich the calculated difference in pixel value is within a predeterminedthreshold as a group (Step S104). In FIG. 8, in a pixel row Li, acalculated difference in pixel value between adjacent pixels in a rangeoccupied by a finger of the hand Yh of the user is within apredetermined threshold. Therefore, this range is set as a group. Suchgrouping is performed in pixel rows other than the pixel row Li. Thedifference between colors of adjacent pixels being within apredetermined threshold has the same meaning as the colors of theadjacent pixels being the same in the threshold range. By the groupingof Step S104, a region with the same color system is distinguished fromthe other regions. In FIG. 8, the region of the hand Yh of the user isdistinguished from the other regions, and thus the contour shape thereofis captured. In addition, in the regions other than the hand Yh of theuser in FIG. 8, a difference in pixel value between adjacent pixels isalso calculated, and a set of image pickup data in which the calculateddifference is in a predetermined threshold is set as a group. FIG. 9 isa diagram schematically illustrating the results of the differencecalculation and grouping in the entire region of the image pickup regionCR. As illustrated in FIG. 9, for example, if an image of a cloud, amountain, or a tree having the same color is picked up in an areapositioned on the left, right, or upper side of the hand Yh, a set ofimage pickup data of the cloud, the mountain or the tree in which acalculated difference is within a predetermined threshold is also set asa group. In the embodiment, when a contour shape captured by theabove-described grouping is less than the size of the hand Yh of theuser imaged by the camera 61, this shape is excluded from an object forshape comparison described below. As a result, a load of calculationrequired for the shape comparison can be reduced.

Next, the image processing unit 160 compares a contour captured by thegrouping of Step S104 during the previous hand recognition processing toa contour captured by the grouping of Step S104 during the present handrecognition processing to determine whether the captured contour ismoved or the contour shape is changed (Step S106). For example, in FIG.8, when the user moves the hand Yh while maintaining the shape of thefingers (as illustrated in Fig.) or when the user bends the thumb, it isdetermined that the captured contour is moved or the contour shape ischanged, and the image processing unit 160 reads a stored hand contourfrom the hand contour shape storing unit 122 of the storing unit 120 asa comparative shape (Step S108). On the other hand, when the capturedcontour is not moved or the contour shape is not changed, there is ahigh possibility that the captured contour is a cloud, a mountain, or atree in FIG. 8 or an indoor table, an indoor chair, or the like duringindoor use. Accordingly, it is determined that the hand recognitionafter Step S108 is unnecessary, and this routine is finished. In thiscase, the user may not move the hand Yh after putting the hand Yh intothe image pickup region CR. However, since there is a movement when theuser puts the hand Yh into the image pickup region CR, YES is determinedin Step S106, and the process proceeds to Step S108. Moreover, thedetermination processing in Step S106 may not be provided, and thestored hand contour which is the comparative shape may be read in StepS108 after the contour capturing in Step S104.

FIG. 10 is a diagram illustrating the summary of hand contours which arestored in the hand contour shape storing unit 122 of the storing unit120 as a comparative shape. As illustrated in FIG. 10, in the handcontour shape storing unit 122, data representing a contour of the handYh where the thumb is open and the index finger is stretched, datarepresenting a contour of the hand Yh where only the index finger isstretched and the other fingers are bent, data representing a contour ofthe hand Yh where the index finger and the middle finger are stretched,and the like are stored. These contours are defined and data regardingthe contours are generated while preliminarily estimating the shapes ofthe hand Yh which can be formed when the user puts the hand into theimage pickup region CR. of the camera 61, and stored in the hand contourshape storing unit 122 in advance. In Step S108 following Step S106, theimage processing unit 160 reads the contour data of the hand Yh of FIG.10 and compares the captured contour of Step S104 to the contourscorresponding to the contour data of the hand Yh of FIG. 10 to determinewhether or not they match (Step S110). The image processing unit 160performs this determination using a method such as an unevennesscomparison method of comparing unevenness statuses. When it isdetermined that the contours match, the image processing unit 160recognizes the determined contour as the hand Yh of the user and setsthe value “1”, which indicates that the hand recognition is completed,to a recognition flag Fr (Step S112) The default value of thisrecognition flag Fr is 0, and 0 is reloaded in the recognition flag Frwhen the head mounted display 100 is turned off or while notificationprocessing described below is executed.

Next, the image processing unit 160 modifies image data for forming thevirtual image VI in cooperation with the AR processing unit 142 and thelike such that a part of the hand Yh of the user recognized in StepS112, for example, the virtual image VI is embedded with a pointer Pcorresponding to a fingertip (Step S114). Due to the modified imagedata, the user visually recognizes the virtual image VI including thepointer P. FIG. 11 is a diagram schematically illustrating a state wherethe virtual image VI is embedded with the pointer P corresponding to thefingertip of the recognized hand Yh of the user. The part of the fingerof the user corresponding to the pointer P in the virtual image VI isnot limited to the fingertip and may be the base of a finger or themiddle of a stretched finger in the hand Yh of the user recognized inStep S112.

The image processing unit 160 calculates coordinates of the fingertip ofthe hand Yh in the image pickup region CR from the rows of the pixelscorresponding to the image pickup data output from the camera 61 andcalculates fingertip coordinates, which are converted from the abovecoordinates when the image pickup region CR is modified according to adisplay rectangle of the virtual image VI, as coordinates of the pointerP. The AR processing unit 142 regenerates (modifies) the image data forforming the virtual image VI such that the pointer P is displayed at theconverted coordinates of the virtual image VI. As a result, asillustrated in FIG. 11, the user visually recognizes the virtual imageVI including the pointer P. Next, whenever the hand Yh of the user movesin the range of the image pickup region CR, the image processing unit160 calculates coordinates of the fingertip of the hand Yh andcalculates converted coordinates of the pointer in the virtual image VI.In response to this, the AR processing unit 142 updates image data fordisplaying the virtual image VI along the movement of the fingertip ofthe hand Yh. Therefore, the AR processing unit 142 allows the user tovisually recognize the virtual image VI while moving the pointer P inthe virtual image VI.

In addition, the image processing unit 160 embeds the virtual image VIwith not only the pointer P but a rectangular frame VIf in an insideperiphery of the display rectangle of the virtual image VI to bevisually recognized by the user. This rectangular frame VIf is displayedto correspond to an outer peripheral region CRf forming an outerperiphery of the image pickup region CR by the AR processing unit 142generating image data thereof. As a result, the user recognizes apositional relationship between the outer periphery of the image pickupregion CR and the hand Yh. The rectangular frame VIf may be displayed atall times or may be triggered to be displayed by a manipulation, forexample, a touch operation of the touch pad 14 by the user.

A-3. Notification Processing

FIG. 12 is a diagram illustrating the summary of notification processingwhich is executed by the control unit 10 including the image processingunit 160 and the AR processing unit 142. As illustrated in FIG. 12,after putting the hand Yh into the image pickup region CR, the usermoves the hand Yh in the image pickup region CR or moves the hand fromthe inside of the image pickup region CR to the outside peripheral sidethereof. Such a movement of the hand is performed by the user having anintention, but the user may not be able to visually recognize the imagepickup region CR of the camera 61. Therefore, the user intends to movein the image pickup region CR; however, contrary to this intention, thefingertip of the hand Yh may pass the outer peripheral region CRfforming the outer periphery of the image pickup region CR, and the handYh may move to the outside of the image pickup region CR. As describedabove using FIG. 11, the fingertip of the hand Yh corresponds to thepointer P. Therefore, when the fingertip of the hand Yh moves up to theoutside of the image pickup region CR, the user cannot recognize thepointer P corresponding to the fingertip included in the virtual imageVI. The same shall be applied to a case where the pointer P correspondsto the base of a finger or the middle of a stretched finger in the handYh of the user. The base of a finger or the middle of a stretched fingermay pass the outer peripheral region CRf forming the outer periphery ofthe image pickup region CR, and the hand Yh may move to the outside ofthe image pickup region CR. In this case, in order for the user to movethe hand Yh in the image pickup region CR, the head mounted display 100according to the embodiment executes the notification processingdescribed below. FIG. 13 is a flowchart illustrating the order of thenotification processing.

This hand recognition processing is repeatedly executed, and the controlunit 10 determines whether or not the value “1” is set to theabove-described recognition flag Fr (Step S210). When it is determinedthat the value “1” is not set to the recognition flag Fr, the hand Yh isnot recognized as described above using FIG. 6, and the hand Yh ispositioned outside the image pickup region CR and does not enter theimage pickup region CR. In this case, there is no case where, asdescribed above using FIG. 12, the hand Yh in the image pickup region CRmoves to the outside of the image pickup region CR such that thefingertip passes the outer peripheral region CRf. Accordingly, when NOis determined in Step S210, this routine is temporarily finished.

On the other hand, when the control unit 10 determines that the value“1” is set to the recognition flag Fr in Step S210, the hand Yh is heldin the image pickup region CR after entering the image pickup region CR.Accordingly, through the following processing, the control unit 10monitors the next behavior of the hand Yh held in the image pickupregion CR. That is, after YES is determined in Step S210, the controlunit 10 calculates fingertip coordinates of the hand Yh in the imagepickup region CR from the rows of the pixels corresponding to the imagepickup data output from the camera 61, detects a fingertip position fromthe fingertip coordinates, and calculates a distance of the fingertipposition from the outermost periphery of the image pickup region CR(Step S220).

Next, the control unit 10 determines whether or not the fingertipposition is in the strip region of the outer peripheral region CRf ofFIGS. 11 and 12 based on the distance of the fingertip positioncalculated from Step S220 (Step S230). When YES is determined in StepS230, the hand Yh held in the image pickup region CR moves to the outerperipheral region CRf. Therefore, the control unit 10 emits an alarmsound from the right earphone 32 and the left earphone 34 so as tonotify the user of the fact that the hand Yh is about to move outsidethe image pickup region CR (Step S240).

FIG. 14 is a diagram illustrating a relationship between behavior of thehand Yh held in the image pickup region CR and the outer peripheralregion CRf and a relationship between the behavior of the hand Yh heldin the image pickup region CR and an emission status of the alarm sound.FIG. 14 illustrates the following. When the hand Yh is held in the imagepickup region CR positioned inside the outer peripheral region CRf, analarm sound is not emitted. As the hand Yh approaches the outermostperiphery of the image pickup region CR after reaching the outerperipheral region CRf, an alarm sound in a high frequency range isemitted. When the hand Yh is out of the image pickup region CR, theemission of the alarm sound which has been emitted in a high frequencyrange is stopped. The relationship between the fingertip position andthe outer peripheral region CRf and the relationship between thefingertip position and the frequency of the emitted alarm sound arestored in the frequency map 124 as a map of frequencies with respect tothe fingertip position. Accordingly, when emitting the alarm sound inStep S240, the control unit 10 emits, with reference to the frequencymap 124, an alarm sound with a frequency corresponding to the distanceof the fingertip position calculated in Step S220. After such an alarmsound is emitted, the process proceeds to Step S220 described above.

In Step S220 following the emission of the alarm sound in Step S240, thedistance of the fingertip position is recalculated. In the next StepS230, whether or not the fingertip position is in the outer peripheralregion CRf is determined. That is, while the fingertip position is inthe strip region of the outer peripheral region CRf, Steps S220 to S240are repeated, and the alarm sound in a frequency range corresponding tothe distance of the fingertip position is continuously emitted. Asillustrated in FIG. 14, as the fingertip position approaches theoutermost periphery of the image pickup region CR, that is, the outerperiphery of the outer peripheral region CRf, the frequency of the alarmsound increases. As a result, this alarm sound notifies the user of thefact that the fingertip position is in the outer peripheral region CRfand thus the hand Yh is about to deviate from the image pickup regionCR.

On the other hand, when it is determined that the fingertip position isnot in the strip region of the outer peripheral region CRf in Step S230,the control unit 10 determines whether or not the fingertip position isshifted outside the outer peripheral region CRf (Step S250). The controlunit 10 stores the distance of the fingertip position calculated in StepS220 in the storing unit 120 or in a specific address in a time-seriesmanner. Therefore, the control unit 10 determines whether or not thefingertip position is shifted outside the outer peripheral region CRfbased on the shift of the distance of the fingertip position. When YESis determined in Step S250, the fingertip position is shifted outsidethe outer peripheral region CRf, that is, the hand Yh is positionedoutside the image pickup region CR. Therefore, the control unit 10resets the recognition flag Fr (Step S270), stops the alarm sound (StepS260), and finishes this routine. When NO is determined in Step S250,the fingertip position is shifted from the outer peripheral region CRfto the inside of the image pickup region CR. Therefore, the control unit10 allows the process to proceed to Step S280 without resetting therecognition flag Fr, stops the alarm sound, and finishes this routine.

In the head mounted display 100 according to the embodiment having theabove-described configurations, when the hand Yh of the user wearing thehead mounted display 100 enters the image pickup region CR of the camera61 which picks up images of the front region of the user as illustratedin FIG. 1 (FIG. 6: Step S112), the behavior of the fingertip of the handYh in the image pickup region CR is monitored based on the image pickupdata output from the camera 61 (FIG. 13: Steps S220 to S230). Moreover,in the head mounted display 100 according to the embodiment, when thehand Yh of the user in the image pickup region CR reaches the outerperipheral region CRf forming the outer periphery of the image pickupregion CR, the alarm sound is emitted to give a notification to the user(Step S240). Accordingly, the following advantageous effects can beobtained.

It is assumed that the user moves and puts the hand Yh into the imagepickup region CR of the camera 61. Next, when the user tries to move thehand Yh from the image pickup region CR to the outside thereof (refer toFIG. 12), the user which moves the hand receives, through the emissionof the alarm sound, the notification that the hand Yh of the user in theimage pickup region CR reaches the outer peripheral region CRf formingthe outer periphery of the image pickup region CR. The user whichreceives this notification can recognize that, if the hand Yh is movedany further, the hand Yh would deviate from the image pickup region CR.Therefore, the user can stop the movement of the hand Yh or can returnthe hand Yh to the detection region such that the hand Yh is held in theimage pickup region CR. As a result, according to the head mounteddisplay 100 of the embodiment, once the hand Yh of the user is detected,this detected state can be continued, which can improve the detectionaccuracy of the hand Yh.

In the head mounted display 100 according to the embodiment, when thevirtual image VI based on the image data generated by the AR processingunit 142 is visually recognized by the user through the image processingunit 160 and the image display unit 20, the virtual image VI is embeddedwith the pointer P corresponding to the fingertip of the detected handYh of the user based on the correspondence between the position of thefingertip position in the image pickup region CR and the position of thepointer P in the display region of the virtual image VI to be visuallyrecognized by the user (Step S114: FIG. 11). At this time, the headmounted display 100 according to the embodiment calculates thecoordinates of the fingertip of the recognized hand Yh of the user inthe image pickup region CR and converts these coordinates into thecoordinates of the pointer P in the virtual image VI. As a result, thepointer P is moved in the virtual image VI along the movement of thehand Yh such that the movement of the hand Yh is associated with themovement of the pointer P to be recognized by the user. Accordingly,according to the head mounted display 100 of the embodiment, the hand Yhof the user can be used for a command operation, and thus conveniencecan be improved. In addition, the notification that the hand Yh isdetected can be given to the user in the state where the behavior of thehand Yh of the user is associated with the behavior of the pointer P ofthe virtual image VI.

In the head mounted display 100 according to the embodiment, when thevirtual image VI based on the image data generated by the AR processingunit 142 is visually recognized by the user, the virtual image VI isembedded with the rectangular frame VIf corresponding to the outerperipheral region CRf forming the outer periphery of the image pickupregion CR and is visually recognized by the user (FIG. 11). Accordingly,according to the head mounted display 100 of the embodiment, theposition of the hand Yh of the user in the image pickup region CR can bereliably recognized by the user. Therefore, the detected state of thehand Yh of the user can be continued and the detection accuracy of thehand Yh can be improved with high effectiveness.

In the head mounted display 100 according to the embodiment, as the handYh of the user inside the image pickup region CR approaches the outerperiphery of the outer peripheral region CRf forming the outer peripheryof the image pickup region CR, the sound with a higher frequency isemitted. Accordingly, while the frequency of the alarm is beingincreased, the alarm sound notifies the user of the fact that thefingertip of the hand Yh is in the outer peripheral region CRf and thusthe hand Yh is about to deviate from the image pickup region CR or thefact that the hand Yh of the user in the image pickup region CR is aboutto move outside the image pickup region CR. As a result, according tothe head mounted display 100 of the embodiment, the user can be urged tohold the hand Yh in the image pickup region CR. Therefore, the detectedstate of the hand Yh of the user can be continued with highereffectiveness.

In the head mounted display 100 according to the embodiment, when thefact that hand Yh of the user is in the image pickup region CR of thecamera is recognized by the user, the contour shape of the hand Yhimaged by the camera 61 is stored in the hand contour shape storing unit122 in advance. Further, in the head mounted display 100 according tothe embodiment, the image pickup data per pixel included in the camera61 is input (Step S100), the difference between colors of adjacentpixels represented by the input image pickup data is calculated (StepS102), and a set of image pickup data having the same color system inwhich the calculated difference in pixel value is within a predeterminedthreshold is set as a group (Step S104: FIGS. 8 to 9). Furthermore, inthe head mounted display 100 according to the embodiment, when thecontour captured by the grouping is compared to the contour shapes ofthe hand Yh stored in the hand contour shape storing unit 122 (StepS110). When the contours match, the contour captured by the grouping isrecognized as the hand Yh of the user in the image pickup region CR.

However, in “Handy AR: Markerless Inspection of Augmented RealityObjects Using Fingertip Tracking” and “The international Journal ofVirtual Reality, 2009, 8(2): 7-12 Robust Hand Tracking Using a SimpleColor Classification Technique”, when a hand of the user imaged by acamera is recognized, the color of the hand is defined in advance as amodel color of the skin, a region of a color which matches with thedefined model color is separated from the other regions, and the shapeof this separated region is recognized as the hand of the user.Accordingly, the recognition accuracy of the hand of the user can besecured to some extent by increasing the model color of the skin definedas the hand of the user or by providing a certain margin to a value ofthe matching determination. However, an image pickup state of the handof the user may vary. For example, there are many cases where theilluminance around the hand of the user may be rapidly changed dependingon changes in outside weather conditions at that time, changes in theirradiation state of the sunlight, the glare of reflected light orshadow, and the like. In such cases, the color of the hand does notmatch with the model color, which may decrease the recognition accuracyof the hand of the user. In a room, the above-described phenomenon mayoccur depending on illumination conditions. In addition, the color ofthe hand of the user varies depending on the race of a person and theindividual people. Therefore, whenever the color of the hand of the usercannot be recognized, it is necessary that the model color, which is thereference for recognizing the hand of the user, be newly definedaccording to the color of the hand of the user. As a result, conveniencedeteriorates. Moreover, even when the model color is newly defined, therecognition accuracy may be decreased due to rapid changes inilluminance.

On the other hand, in the head mounted display 100 according to theembodiment, as described above, the contour is captured by the groupingbased on the calculation of the color difference between adjacentpixels, and the captured contour is compared to the stored contourshapes of the hand Yh, thereby allowing the user to recognize the handYh. Therefore, the following advantageous effects can be obtained. Thatis, according to the head mounted display 100 of the embodiment, it isnot necessary that the model color which is the reference forrecognizing the hand of the user be defined for each color of the hand.Therefore, when the hand Yh of the user is recognized, convenience doesnot deteriorate. In addition, even when the illuminance around the handof the user is rapidly changed, the respectively pixels aresubstantially equally affected by the rapid change during thecalculation of the color difference between adjacent pixels. Therefore,the color difference between adjacent pixels is not affected that muchby the rapid change in the illuminance around the hand Yh of the user.Accordingly, according to the head mounted display 100 of theembodiment, a decrease in the recognition accuracy of the hand by therapid change in the illuminance is suppressed, and the recognitionaccuracy can be improved. In addition, the contour shapes of the handstored in the hand contour shape storing unit 122 are not particularlylimited as long as they are expected to be used in the image pickupregion CR by the user for a purpose. It is not necessary that the storedcontour shapes of the hand be limited to some extent and be newly setdepending on the race of a person and the individual users. From thispoint of view, according to the head mounted display 100 of theembodiment, when the hand Yh of the user is recognized, convenience doesnot deteriorate, and the versatility of the technique for detecting ahand Yh of the user is improved, and the cost thereof can be reduced.

In the head mounted display 100 according to the embodiment, in a casewhere the contour captured by the grouping is compared to the contourshapes of the hand Yh stored in the hand contour shape storing unit 122,when the captured contour moves in a predetermined shape change range orwhen the shape of the captured contour is changed (YES in Step S106),the captured contour is compared to the stored contour shapes of thehand. As a result, the following advantageous effects can be obtained.When the camera picks up images of the front region of the user, thecamera may pick up images of other objects in addition to the hand ofthe user. For example, when the camera confronting the user picks upimages of the front region of the user, so-called still images of notonly the hand Yh of the user but the face and the upper body of theuser, an indoor table and an indoor chair positioned behind the user, orthe like is picked up. In addition, the camera 61 equipped in the imagedisplay unit 20 picks up still images of a cloud, a mountain, or a treein FIG. 8 or an indoor table, an indoor chair, or the like during indooruse. In the case of these still images, contours thereof are not movedand contour shapes are not changed. On the other hand, typically, theuser puts the hand Yh into the image pickup region CR for some purposes.Therefore, the hand Yh may be moved in the image pickup region or theshape thereof may be changed without a significant change in position.Accordingly, even when still images of a cloud, a mountain, or a tree inFIG. 8 or an indoor table, an indoor chair, or the like during indooruse are picked up, contours are not moved and contour shapes are notchanged. Therefore, the captured contours of the still images can beexcluded from the comparison with the stored contour shapes of the hand.Accordingly, according to the head mounted display 100 of theembodiment, a load of calculation required for the shape comparison canbe reduced.

In the head mounted display 100 according to the embodiment, in aportion where the virtual image VI including the pointer P is displayed,the user visually recognizes the outside scene SC, which is positionedbehind the virtual image VI, through the virtual image VI. Accordingly,the virtual image VI is visually recognized by the user so as to bedisplayed on the image pickup region CR of the camera 61 through thehand Yh of the user. As a result, according to the head mounted display100 of the embodiment, the hand Yh of the user and the virtual image VIwhich overlap each other are recognized by the user. Therefore, therecognition of the pointer P with respect to the operation of the handYh of the user can be improved.

In the head mounted display 100 according to the embodiment, when thescanning input of the image pickup data is received, the movement of thehead of the user is cancelled based on sensor outputs from the 9-axissensor 66. Accordingly, according to the head mounted display 100 of theembodiment, the contour having the same color system obtained by thegrouping can be accurately captured without depending on the movement ofthe head of the user. Therefore, the recognition accuracy of the hand ofthe user can be increased.

A-4. Other Embodiment-1

The head mounted display 100 can be embodied as follows. FIG. 15 is adiagram schematically illustrating hand recognition processing which isexecuted by a head mounted display 100 according to another embodimentof the invention. In this embodiment, the scanning input of the imagepickup data in Step S100 and the difference calculation in Step S102subsequent thereto are executed in the execution order determined foreach divided region into which the image pickup region CR is divided inadvance. That is, as illustrated in FIG. 15, the image pickup region CRis divided into a left end region CRL, a right end region CRR, a lowerend region CRD, and a residual region CRU in advance, and the scanninginput of data in the horizontal direction for each divided region isexecuted in the following order: the right end region CRR→the lower endregion CRD→the left end region CRL. Typically, the user puts thedominant hand Yh into the image pickup region CR of the camera on theside seen from the user. Therefore, when the dominant hand Yh is theright hand, the user puts the hand Yh from the right end region CRR orfrom the lower end region CRD to the image pickup region CR. In theabove-described embodiment, since the fingertip matches with the pointerP, even the user with the left dominant hand is expected to perform thesame operation as the user with the right dominant hand. Accordingly,according to the head mounted display 100 having the configuration, theimage pickup region CR is divided into the left end region CRL, theright end region CRR, the lower end region CRD, and the residual regionCRU in advance, and the scanning input of data is executed in theabove-described order. As a result, by associating the execution orderdetermined for each divided region with the operation of putting thehand Yh into the image pickup region CR, the hand Yh can be rapidlyrecognized when the hand Yh of the user enters the right end region CRR.Then, the behavior of the hand Yh can be associated with the behavior ofthe pointer P by the data input in the entire region of the image pickupregion CR. In the embodiment, the scanning input of data may be executedin the following order: the lower end region CRD→the right end regionCRR→the left end region CRL. Alternatively, the scanning input of datamay be executed in the following order: the left end region CRL→thelower end region CRD→the right end region CRR, and the differencecalculation may be executed first in any region of the left end regionCRL, the lower end region CRD, and the right end region CRR.

A-5. Other Embodiment-2

In a head mounted display 100 according to another embodiment of theinvention, RGB-based image pickup data to be input in Step S100 iscolor-converted into HSV color-based data in which the color isrepresented by hue, saturation, and value. The color differencecalculation between adjacent pixels in the next step S102 is replacedwith calculation of a difference between values of adjacent pixelsobtained by the color conversion. The value obtained by the colorconversion of the image pickup data is separated from the brightnessaffected by the illuminance around the imaged hand Yh of the user duringthe color conversion into the HSV color system. Therefore, according tothe head mounted display 100 of the embodiment using the value obtainedby the color conversion into the HSV color system, a decrease in therecognition accuracy by the rapid change in illuminance can besuppressed with higher effectiveness, and the recognition accuracy ofthe hand Yh can be further increased.

A-6. Other Embodiment-3

FIG. 16 is a diagram illustrating a schematic configuration of a headmounted display 100 according to another embodiment of the invention. Inthis embodiment, the above-described camera 61 is used only for pickingup images of the outside scene SC, and a camera 61 a used only forrecognizing the hand Yh of the user is further provided at the end ER ofthe image display unit 20. FIG. 17 is a diagram schematicallyillustrating a relationship between an image pickup region CR of thecamera 61 a, which is positioned at the end ER of the image display unit20, and a visual field VR of the user. As illustrated in FIG. 17, theimage pickup region CR of the camera 61 a deviates from a region on theright end side of the visual field VR of the user or from the visualfield VR due to the camera installation position. Therefore, when thehand Yh in the image pickup region CR does not substantially enter theoutside scene SC in the visual field VR, and the outside scene SC doesnot substantially block the hand Yh. Therefore, the user can view andenjoy the outside scene SC without being disturbed by the hand Yh. Thecamera 61 a may be provided at the end EL or may face downward such thatthe image pickup region CR is positioned below the visual field VR ofthe user. The head mounted display 100 according to the embodimentnotifies the user as follows of the fact that the hand Yh of the userenters the image pickup region CR of the camera 61 a. FIG. 18 is adiagram illustrating the summary of a method of detecting the hand whenthe image pickup region CR of the camera 61 a deviates from the visualfield VR of the user.

As illustrated in FIGS. 18A and 18B, in the embodiment, when the hand Yhwhich has been positioned outside the image pickup region CR of thecamera 61 a enters the image pickup region CR, the hand Yh of the useris recognized thorough the hand recognition described using FIG. 6 (StepS112). As a result, the AR processing unit 142 generates image data fordisplaying the virtual image VI, which indicates that the hand Yh entersthe image pickup region CR and is recognized, as illustrated in thelower section of FIG. 18 and displays the virtual image VI on an upperright region of the visual field VR based on the image data. As aresult, the user can recognize that the hand Yh which has beenpositioned outside the image pickup region CR of the camera 61 a entersthe image pickup region CR. Therefore, the behavior of the hand can beconsidered a command operation. For example, when the user moves thehand Yh after the virtual image VI is displayed as described in thelower section of FIG. 18, the control unit 10 detects this movement. Asa result, the virtual image VI illustrated in FIG. 11 and including thepointer P and the rectangular frame Vif can be displayed, and a commandcan be executed using the pointer P. The virtual image VI illustrated inFIG. 18 may be displayed together with the rectangular virtual image VIillustrated in FIG. 11, or may be displayed to include the rectangularvirtual image VI.

B. Modification Example

In the above-described embodiments, a part of configurations which areimplemented by hardware may be implemented by software. Conversely,apart of configurations which are implemented by software may beimplemented by hardware. In addition, the following modifications can bemade.

Modification Example 1

In the above-described embodiments, the configurations of the headmounted display have been described as an example. However, theconfigurations of the head mounted display can be arbitrarily madewithin a range not departing from the scope of the invention. Forexample, addition, deletion, conversion, and the like can be made foreach component.

In the above-described embodiments, the assignment of the components tothe control unit and the image display unit is merely exemplary, andvarious forms can be adopted for the assignment. For example, thefollowing forms can be adopted: (i) a form in which the control unit hasprocessing functions of the CPU, the memory, and the like and the imagedisplay unit has only the display function; (ii) a form in which boththe control unit and the image display unit have processing functions ofthe CPU, the memory, and the like; (iii) a form in which the controlunit and the image display unit are integrated (for example, a form inwhich the image display unit includes the control unit and functions asa wearable computer); (iv) a form in which a smartphone or a portablegame machine is used instead of the control unit; and (v) a form inwhich the connecting unit (cord) is not provided by configuring thecontrol unit and the image display unit to be capable of radiocommunication and wireless power supply.

In the above-described embodiments, for convenience of explanation, thecontrol unit includes the transmitting units, and the image display unitincludes the receiving units. However, both of the transmitting unitsand the receiving units in the above-described embodiments have afunction for enabling bidirectional communication and can function astransmitting and receiving units. In addition, for example, the controlunit illustrated in FIG. 2 is connected to the image display unitthrough a wired signal transmission line. However, the control unit andthe image display unit may be connected through a wireless signaltransmission line such as a wireless LAN, infrared communication, orBluetooth (registered trademark).

For example, the configurations of the control unit and the imagedisplay unit illustrated in FIG. 2 can be arbitrarily changed.Specifically, for example, the touch pad may be removed from the controlunit and the control unit may be configured to be operated by only thecross key. In addition, the control unit may include anothermanipulation interface such as a manipulation stick. The control unitmay be configured to be connected with devices such as a keyboard or amouse. The control unit may receive an input from the keyboard and themouse. In addition, for example, a manipulation input may be acquirednot only from the touch pad and the cross key but from a foot switch(manipulation switch using a foot of the user). In addition, if amanipulation input can be acquired from the foot switch or from thevisual line of the user, the input-information acquiring unit canacquire the manipulation input from the user during an operation inwhich it is difficult for the user to use the hands.

For example, the head mounted display is the transmissive head mounteddisplay of the binocular type. However, the head mounted display may bea head mounted display of a monocular type. The head mounted display maybe configured as a non-transmissive head mounted display in whichtransmission of an outside scene is blocked in a state in which the userwears the head mounted display.

FIGS. 19A and 19B are diagrams illustrating a configuration of anexternal appearance of a head mounted display according to amodification example of the invention. An example of FIG. 19A isdifferent from the head mounted display 100 illustrated in FIG. 1, inthat an image display unit 20 a includes a right optical-image displayunit 26 a instead of the right optical-image display unit 26 andincludes a left optical-image display unit 28 a instead of the leftoptical-image display unit 28. The right optical-image display unit 26 ais formed to be smaller than the optical members illustrated in FIG. 1and is arranged obliquely above the right eye when the user wears thehead mounted display. Likewise, the left optical-image display unit 28 ais formed to be smaller than the optical members illustrated in FIG. 1and is arranged obliquely above the left eye when the user wears thehead mounted display. An example of FIG. 19B is different from the headmounted display 100 illustrated in FIG. 1, in that an image display unit20 b includes a right optical-image display unit 26 b instead of theright optical-image display unit 26 and includes a left optical-imagedisplay unit 28 b instead of the left optical-image display unit 28. Theright optical-image display unit 26 b is formed to be smaller than theoptical members illustrated in FIG. 1 and is arranged obliquely belowthe right eye when the user wears the head mounted display. Likewise,the left optical-image display unit 28 b is formed to be smaller thanthe optical members illustrated in FIG. 1 and is arranged obliquelybelow the left eye when the user wears the head mounted display. In thisway, the optical-image display units only need to be arranged near theeyes of the user. In addition, the size of the optical members whichform the optical-image display units is arbitrary. A head mounteddisplay having a configuration in which the optical-image display unitscover only a part of the eyes of the user, that is, a configuration inwhich the optical-image display units do not completely cover the eyesof the user may also be implemented.

For example, the functional units such as the image processing unit, thedisplay control unit, the AR processing unit, and the sound processingunit have been described as being implemented by the CPU loading thecomputer program, which is stored in the ROM or the hard disk, on theRAM and executing the computer program. However, the functional unitsmay be configured using an Application Specific Integrated Circuit(ASIC) designed to implement the functions of the functional units.

For example, in the above-described embodiments, the image display unitof the head mounted display is worn like eyeglasses. However, the imagedisplay unit may be a normal flat display apparatus (for example, aliquid crystal display apparatus, a plasma display apparatus, or anorganic EL display apparatus). In this case, the control unit and theimage display unit may be connected through a wired signal transmissionline or may be connected through a wireless signal transmission line.With such a configuration, the control unit can be used as a remotecontroller for the normal flat display apparatus.

Instead of the image display unit worn like eyeglasses, an image displayunit having another shape such as an image display unit worn like a capmay be adopted as the image display unit. Further, an ear-hang type or aheadband type may be adopted as the earphones, or the earphones may notbe provided. In addition, for example, the head mounted display may beconfigured as a head-up display (HUD) which is mounted to vehicles suchas automobiles or airplanes. In addition, for example, a head mounteddisplay equipped with a body protector such as a helmet may also beadopted.

For example, in the above-described embodiments, the secondary batteryis used as the power supply. However, the power supply is not limited tothe secondary battery, and various batteries can be used. For example, aprimary battery, a fuel battery, a solar battery, a thermal battery, andthe like may be used.

In the above-described embodiments, for example, the image-lightgenerating unit is configured using the backlights, the backlightcontrol units, the LCDs, and the LCD control units.

However, the above-described configuration is merely exemplary. Theimage-light generating unit may include components for implementinganother system together with these components or instead of thesecomponents. For example, the image-light generating unit may include anorganic Electro-Luminescence (EL) display and an organic-EL controlunit. In addition, for example, the image-light generating unit may usea digital micromirror device instead of LCD. In addition, for example,the invention can also be applied to a head mounted display apparatus ofa laser retinal projection type.

Other Modification Examples

In the above-described embodiments, as illustrated in FIG. 14, as thehand Yh in the image pickup region CR approaches the outermost peripheryof the image pickup region CR after reaching the outer peripheral regionCRf, the alarm sound in a higher frequency range is emitted. However,the invention is not limited to this configuration. For example, as thehand Yh approaches the outermost periphery of the image pickup region CRafter reaching the outer peripheral region CRf, an alarm sound with ahigher volume is emitted. In addition, a pulse-like alarm sound may beemitted, and as the hand Yh approaches the outermost periphery of theimage pickup region CR after reaching the outer peripheral region CRf,the pulse-like alarm sound with a shorter period may be emitted. A voiceguide such as “the hand is about to deviate from the recognition range”may be announced. In addition, the notification may be given using,instead of sound, light emission or osseous conduction in which a soundis recognized by conducting vibration to the middle ear. In the case ofthe notification using the light emission, as the hand Yh approaches theoutermost periphery of the image pickup region CR, light rays havingdifferent colors may be emitted. Alternatively, as the hand Yhapproaches the outermost periphery of the image pickup region CR afterreaching the outer peripheral region CRf, a pulse light with a shorterperiod may be emitted. In the case of the notification using the osseousconduction, as the hand Yh approaches the outermost periphery of theimage pickup region CR, a sound with a higher frequency or a sound witha higher volume may be conducted to the middle ear. In addition, thevirtual image VI representing a level gauge may be displayed by the ARprocessing unit 142 and the like, and as the hand Yh approaches theoutermost periphery of the image pickup region CR, the level may beincreased. As the hand Yh approaches the outermost periphery of theimage pickup region CR, the pointer P of the virtual image VI may blink,or the color or shape of the pointer P may be changed. In addition, forthe notification, the user can use a device which is provided separatelyfrom the head mounted display 100. For example, a control signal isoutput to the control unit 10 (refer to FIG. 1) and a vibration inducingunit and the like included in a portable terminal (not illustrated),vibration is induced by the control unit 10 and the portable terminaland the like, and a notification is given to the user by this vibration.In this case, as the hand Yh approaches the outermost periphery of theimage pickup region CR after reaching the image pickup region CR,vibration with a shorter period may be induced by the portable terminaland the like. The image display unit 20 itself worn on the head mayinduce vibration.

In the head mounted displays 100 according to the above-describedembodiments, unless the hand Yh of the user is recognized as illustratedin FIG. 6 within a predetermined period, only the rectangular frame VIfillustrated in FIG. 11 or the virtual image VI including the rectangularframe VIf may be recognized by the user through the AR processing unit142 and the like. As a result, the user can reliably recognize the factthat the hand Yh of the user does not enter the image pickup region CRand can be urged to put the hand Yh into the image pickup region CRpositioned inside the outer peripheral region CRf. Therefore, thedetection accuracy of the hand Yh of the user is increased. Only therectangular frame VIf illustrated in FIG. 11 or the virtual image VIincluding the rectangular frame VIf may be triggered to be displayed bya manipulation, for example, a touch manipulation of the user on thetouch pad 14.

In addition, in the AR processing unit, an outside scene image obtainedby the camera in the visual field direction of the user may be underpattern-matching using a pixel parallactic angle to implement theaugmented-reality processing. Specifically, the image display unitincludes a right eye camera and a left eye camera. The right eye camerais arranged at a position of the image display unit corresponding to theright eye of the user so as to pick up images of an outside scene in thefront direction of the image display unit. The left eye camera isarranged at a position of the image display unit corresponding to theleft eye of the user so as to pick up an image of an outside scene inthe front direction of the image display unit. The AR processing unitmay calculate a deviation amount between a target object (which is atarget for displaying additional presentation information around thetarget) included in the image picked up by the right eye camera and atarget object included in the image picked up by the left eye camera andmay determine “a target distance”, which is a display position of thevirtual image VI in the augmented-reality processing, using thedeviation amount and the pixel parallactic angle.

The AR processing unit may execute the above-described augmented-realityprocessing only when predetermined conditions are satisfied. Forexample, the AR processing unit may execute the above-describedaugmented-reality processing only when the image display unit has aconfiguration in which a visual line direction of the user can bedetected, and when the detected visual line direction satisfies at leastone of the following conditions.

In a viewing angle range in which a horizontal angle is 200° and avertical angle is 125° (an angle in the down direction is 75° and anangle in the up direction is 50°)

In a viewing angle range in which a horizontal angle is 30° and avertical angle is 20°, which is an effective visual field where aninformation receiving capability is superior

In a viewing angle range in which a horizontal angle is 60° to 90° and avertical angle is 45° to 70°, which is a stable point-of-regard fieldwhere a point of regard is rapidly and stably observed

In a viewing angle range from a horizontal angle of 20°, at whichself-motion perception (vection) starts to be induced on an image, to ahorizontal angle of 110° at which self-motion perception is saturated

In addition, in the above-described embodiments, the camera 61 includingthe imaging device is used to recognize the hand Yh of the user.However, the image display unit 20 worn on the head may be provided withan ultrasonic sensor, an infrared sensor, an optical sensor, or the likesuch that the hand of the user can be recognized based on the sensoroutputs. Therefore, even when the recognized hand is about to deviatefrom the image pickup region CR of the camera 61, the notification canbe given to the user as described above.

In addition, in the head mounted displays 100 according to theabove-described embodiments, when the hand Yh of the user in the imagepickup region CR is recognized, the calculation of the difference inpixel value between adjacent pixels to which data is input is executedper pixel included in the camera 61 (Step S102). However, in aperipheral region including a picked-up image of a contour capturingtarget object which is picked up in the image pickup region CR, thecalculation of the difference in pixel value between adjacent pixels maybe executed. Alternatively, a difference calculation of data may beperformed on colors obtained from pixels in the above peripheral region.In the difference calculation on pixels in the above peripheral region,still images of a cloud, a mountain, a tree, and the like illustrated inFIG. 8 can be excluded from the contour shape capturing target whichundergoes the difference calculation. Accordingly, a calculation load isreduced.

The invention is not limited to the above-described embodiments,examples, and modification examples, and various configurations can bemade within a range not departing from the scope of the invention. Forexample, the technical features of the embodiments, the examples, andthe modification examples, which correspond to the technical features ofeach configuration described in “SUMMARY”, can be appropriatelysubstituted or combined with other features to solve a part or all theabove-described problems or to achieve a part or all the above-describedeffects. In addition, unless described as being essential in thisspecification, these technical features may be appropriately deleted.

The entire disclosure of Japanese Patent Application No. 2013-177866,filed Aug. 29, 2013 is expressly incorporated by reference herein.

What is claimed is:
 1. A head mounted display apparatus in which avirtual image and an outside scene overlap each other and can bevisually recognized by a user, the apparatus comprising: anaugmented-reality processing unit that generates image data fordisplaying the virtual image and allows the user to visually recognizethe virtual image such that the virtual image is displayed in a visualfield of the user based on the generated image data; a detecting unitthat detects a hand of the user in a predetermined detection region fordetecting the hand of the user and detects behavior of the hand of theuser in the detection region; and a notifying unit that gives anotification to the user based on the behavior of the hand of the userdetected by the detecting unit when the hand of the user reaches anouter peripheral region forming an outer periphery of the detectionregion.
 2. The head mounted display apparatus according to claim 1,wherein the augmented-reality processing unit allows the user tovisually recognize the virtual image in a display region correspondingto the detection region, and when the detecting unit detects the hand ofthe user in the detection region, the augmented-reality processing unitgenerates the image data for displaying the virtual image, which isembedded with a partial hand image corresponding to at least a part ofthe detected hand of the user, and updates a position of the partialhand image in the display region of the virtual image based on aposition of the part of the hand in the detection region.
 3. The headmounted display apparatus according to claim 1, wherein theaugmented-reality processing unit generates the image data fordisplaying the virtual image, which is embedded with a frame shapecorresponding to the outer peripheral region forming the outer peripheryof the detection region, to allow the user to visually recognize thevirtual image.
 4. The head mounted display apparatus according to claim1, wherein when the detecting unit detects the hand of the user in thedetection region, the augmented-reality processing unit generates theimage data for displaying a detection-completed image, which indicatesthe completion of the detection of the hand of the user, as the virtualimage or generates the image data for displaying the virtual image,which is embedded with the detection-completed image, to allow the userto visually recognize the virtual image.
 5. The head mounted displayapparatus according to claim 1, wherein the notifying unit is configuredto change a form of the notification to the user and, as the hand of theuser in the detection region approaches the outer periphery of the outerperiphery region, gives the notification to the user while changing theform of the notification.
 6. The head mounted display apparatusaccording to claim 5, wherein as the hand of the user in the detectionregion approaches the outer periphery of the outer periphery region, thenotifying unit emits a sound with a higher frequency or emits a soundwith a higher volume to give the notification to the user.